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ABSTRACT 

In response to the findings and recommendations of a 
planning conference in 1989, the Department of Energy (DOE) increased 
its funding of ongoing precollege programs in science and mathematics 
education and launched some new initiatives. To evaluate and improve 
its efforts, the DOE has also initiated a 4-year evaluation and 
technical assistance project at the National Center for Improving 
Science Education. This paper focuses on the strategies developed and 
some results emerging from the formative evaluation component of this 
project. The major formative evaluation strategy is the profiling of 
the doe's education activities, with the development of a template 
for each of the five identified program types. The template for 
profiling the teacher-development programs of the DOE laboratories is 
presented as an example of the way in which a template can be used in 
formative evaluation. Some of the emerging results of the 
teacher-development research are outlined to demonstrate evaluation 
results. One figure illustrates an excerpt from the teacher 
development template. (Contains 5 references.) (SLD) 
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Background 

In 1989, the U.S. Department of Energy (DOE) held a planning conference in Berkeley 
on how it could respond effectively to the challenge presented by The Nation at Risk, the 
document that proclaimed in 1983 that "the educational foundations of our society are 
presently bemg eroded by a rising tide of mediocrity that threatens our very future as a 
Nation and as a people." The report resulting from the Berkeley conference emphasized 
DOE'S mission of supplying the nation with the "energy resources, technologies, and 
information needed for economic progress and national security," and the need to build 
"a scientifically and technologically informed citizenry to help chart a course for public 
energy policy." It also concluded that DOE would have difficulty in realizing its mission 
because the low achievement of many students in mathematics and science kept them 
from advancing sufficiently to consider technical occupations. One of the major 
problems noted in the report was the inadequate training of teachers; other problems 
-oted were outdated curricula, poor facilities and equipment for teaching science, and 
uck of student interest and motivation, particularly on part of ethnic minorities and girls. 
Because of its special facilities and its core of highly qualified personnel, the DOE was 
thought to be strategically positioned to begin addressing some of these problems. 

In response to the findings and recommendation of the Berkeley conference, the DOE 
substantially increased its funding of ongoing precollege programs in science and 
mathematics education and launched several new initiatives. In 1990, over 200,000 
(V) teachers and students were affected by DOE programs; the agency's 1993 budget 
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allocated roughly $25 million to precollege education, not counting contributions by the 
Laboratories. This level of effort signals considerable involvement of DOE's science and 
engineering community working to improve science learning in the schools. Aware of 
the need to evaluate and improve its efforts, the DOE also has funded a four-year 
evaluation and technical assistance project at the National Center for Improving Science 
Education. The project is designed to achieve two overarching goals: 

• To develop and initiate a system for ongoing formative and summative evaluation 
of the Department's precollege education programs that will provide feed-back 
loops for future program planning in this area; 

• To work collaboratively with staff from DOE Headquarter, the Laboratories, and 
associated facilities to share knowledge of exemplary science and mathematics 
education and build evaluation capacity within the Department and its internal 
and external program managers. 

These goals are highly interrelated; they are aimed at creating the kinds of self- 
improvement norms and activities needed for programs to achieve their maximum 
impact. We at the Center are taking initial responsibility for a set of research activities 
that will model useful evaluation practices and begin to provide data for what works to 
improve science and mathematics education. These activities are being conducted 
collaboratively with DOE staff and the Ubs, with one of the purposes being to build the 
capacity of program managers who have primary responsibility for formative evaluation 
to continue such activities in the future in order to improve their programs. Although 
the project as a whole is concerned with impact assessment as well as with formative 
evaluation and technical assistance, this paper focuses on the strategies developed and 
some results emerging from the formative evaluation component of the project. 

First, however, we should note the challenge posed by combing in one project evaluation 
activities with technical assistance. For technical assistance to be effective, clients and 
program participants must feel free to reveal problems they are experiencing, be willing 
to have weaknesses uncovered through evaluative infonnation, and have sufficient trust 
to ask hard questions about program elements that don't seem to be working as 
expected. These elements of trust and willingness to share information, even when they 
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point up weaknesses, also are indispensable to formative evaluations which, after all, 
should aim for program improvement. This trust is not easy to maintain when im.pact 
assessment is involved, however, particularly if there is any threat (perceived or real) of 
funding decisions being linked to impact results. Our responses to this tension are as 
follows: 

• We see our role as primarily in the technical assistance and formative evaluation 
modes. This allows us, even as we collect information appropriate for preliminary 
summative evaluations, to accompany such information with explanations of 
salient program and contextual factors and, more importantly, with specific 
recommendations (framed collaboratively with the Labs and DOE headquarters 
staff) for improving programs. 

• Through collaborative activities that we are sponsoring, the Labs are sharing 
information with each other which we believe will serve to make each Lab's 
individual programs more effective. At the same time, we are working with DOE 
headquarters staff to use evaluative information constructively for program 
improvement. 

• We are encouraging DOE to construct a long-range time schedule for the phasing 
of formative and summative evaluations. At any given time, we see both as 
necessary, but in different mixes. In the Ci'rrent initial period, much more energy 
and resources should be devoted to efforts that will improve programs (formative 
evaluation and technical assistance); over the long range (five to ten years, 
depending on the nature of the program), as programs become well defined and 
established, evaluation efforts should concentrate on summative evaluation. 

• Lastly, through a series of seminars with evaluation experts, we are reviewing the 
hallmarks of good evaluations with the DOE participants and engaging them in 
developing criteria for assessing the evaluation information being generated by 
themselves and by outside evaluators, so that they will become informed 
evaluation sponsors and consumers. 
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Formative Evaluation Strategies 



In concert with the priorities set by the Federal Council for Science, Engineering, and 
Technology (FCCSET), the Department of Energy is pursuing two overarching long-term 
poals in its efforts to improve mathematics and science education: 

. Arming teachers with a better grasp of subject matter and more effective 
strategies for teaching science and mathematics through teacher enhancement 
programs; and 

. Improving student outcomes, particularly their achievement and persistence in 
pursuing technical fields. 

In order to achieve these goals, DOE emphasizes the following elements in its programs: 
maximizing the use of the unique resources of the DOE's Laboratories, including the 
science and engineering expertise of their staff, their advanced facilities and equipment, 
and their involvement in cutting-edge research; forming partnerships with other federal 
agencies, states, businesses, higher education institutions, and community organizations; 
focusing on minorities and other groups underrepresented in the science and engineering 
professions; supporting systemic reform, i.e., impact on whole schools, districts, or states; 
and promoting the best in science education research and practice, including emphasis on 
hands-on, experiential learning. 

In the first phase of our work with DOE, we jointly identified five program types: (1) 
intensive experiences for students; (2) systemic change efforts generally involving whole 
districts; (3) research opportunities for teachers; (4) teacher enhancement through 
workshops or institute-like, experiences; and (5) interventions involving students and 
teachers simultaneousb . The existence of programs focused on five differem areas 
increases the challenge of understanding what is occurring with DOE support. During 
the four years of the project's life, we will be concerned with piloting evaluation 
approaches and instruments for each of these different program types. 
Our formative evaluation strategies are designed to find out: 
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. what the primary components of a program are, and which of these the program 
operators believe to be critical; 



. to what extent the design of each program being offered nationally or by 
individual DOE Laboratories matches what we know from research and best 
practice to be effective science and/or mathematics education; 

. to what extent the design matches research findings and best practice in student 
enhancement and motivation, teacher enhancement, or systemic change 
(depending on the type of program); 

. to what extent what is actually happening in the program matches the program 
design, and what facilitating or impeding factors are being encountered; and 

. what needs to be done to improve, stabilize, and institutionalize programs, once 
judgments on effectiveness have been made. 

Not only is it helpful to ask these kinds of questions early in the development of 
promising new education initiatives, but it is also important to build into all parts of such 
programs the capacity and expectations for ongoing reflection and evaluation. In our 
view, the early, formative stages of program development and implementation - which 
characterize many of the current DOE programs - are opportune for such reflective 
activity. 

Our major formative evaluation strategy is the profiling of DOE education activities, i.e., 
carefully gathering information about what is happening when the program is going on. 
This process, developed by the NETWORK over ten years ago (see references), 
generates detailed descriptions and facilitates understanding of how each program works 
and to what extent the programs are actually doing what they are expected to be domg -- 
by DOE headquarters. Lab program developers, and participants. It thus enables 
program managers and evaluators alike to address both implementation and impact 
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We are developing a template for each of the five different program types, based on 
research and exemplary practice where they exist, and enhanced by the experiences of 
program developers as to what makes a particular type of program effective. The use of 
templates facilitates communication, assessment of variation, and understanding of why 
different outcomes occur in different settings. Each template is a description of the 
components of an idealized program model, with possible variations. The template allow 
us to profile each program that within a program type in a standardized way, so that 
similar programs can be compared to the idealized model and within and across settings. 
To date, three templates have been completed: for teacher research participation, for 
teacher enhancement, and for studem programs. The template for systemic development 
is nearing completion. 

Practice profiles and the templates on which they are based typically are developed by 
evaluators in collaboration with program developers. They are piloted in the field to 
check whether any components are missing and whether the full range of variation has 
been amicipated by the developers. The distinction between "ideal", "acceptable", and 
"unacceptable" variations is based largely on the principles of effective practice, the 
tested or theor>'-based concepts underlying the program's design, identified by the 
program's developers. The use of profiles in evaluation can yield quantitative data about 
the extent of miplementation of program components. The variations within each 
component can be scaled from ideal implementation to non-implementation; components 
can be weighted by their developers as to their importance; and summary 
implementation scores can be derived from individual program participants. The degree 
of implememation can be considered when examining im.pact data so that questions 
about why impact seems to be particularly high or low can be addressed. 

Following this methodology, we have worked closely with Lab program and evaluation 
staff and DOE headquarters staff to identify the key components of the programs for 
which we have constructed templates; we have piloted the templates in a limited number 
of settings and revised them on the basis of the pilots; and we have the trained Lab staff 
to use revised templates to profile their programs. We supplement the descriptive 
information about program components gathered through use of the template with data 
on the various structural and cultural elements existing in each setting that serve to 
constrain or support the program. This information begins to address questions of how 
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different settings contribute to the way a program is designed and implemented, and 
ultimately to its success. We hope that for those programs not included in the five 
clusters noted above, Lab staff will be able to develop their own templates as a result of 
our work with them. 

Profiling of the Labs' Teacher Development Programs 

The template for this cluster of programs was developed in early 1992 and piloted in five 
Labs between mid 1992 and early 1993. The remaining Labs then profiled their own 
programs; the information s immarized below is based on data gathered by the National 
Center and the Laboratories. 

One of the seven components of the teacher development template is displayed in detail 
in Figure 1; the other components-spelled out in equal detail-- are: Program 
Administration, Vision for the Classroom, Unique Contribution of the Laboratory, 
Follow-up, Teacher Leadership and Responsibility, and Program Evaluation. The 
complete template, operational definitions, and other information is available from the 
National Center in "Profiling Teacher Development Programs: An Approach to 
Formative Evaluation". The template was used to address questions such as: 

• What is best practice for teacher development programs? 

• To what extent is a program in a specific Lab designed to reflect what is most 
effective? 

• To what extent does the program in place reflect best practice? 

• To what extent is the program's design actually carried out? 

• Where are the gaps? What can be improved? What is beyond the program 
operators' control? 

Five teacher development programs were selected for piloting, which ran from July, 1992 
to April, 1993. They were chosen to vary across such dimensions as grade level of 
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teachers, forrnat, types of target populations, and the scientific discipline focused on. 
The programs and sponsoring Labs were: the Topics in Modem Physics (TMP) program 
of Fermilab; the Science Alive program of Pacific Northwest Laboratory; the Science 
Advisors (SCIAD) program of Sandia National Laboratories, New Mexico; the Teacher 
Opportunities to Promote Science (TOPS) program of Los Alamos National Laboratory; 
and the Chicago Science Explorers program of Argonne National Laboratory. 

The data used for profiling were collected through examina*-'-^" of program materials, a 
two-day site visit by at least two National Center staff and associates, and comments on 
the draft profile by the Lab program staff. The site visit involved interviews with 
program managers, participating teachers, scientists, and in some cases, teachers from 
pa<:i years, students, principals of participating teachers, program evaluators, and partners 
such as school district and museum personnel. In addition, site visitors observed the 
teacher development activities whenever possible. These activities took place in the 
Laboratories, schools and classrooms, hotel meeting rooms, and in the field (e.g., on a 
farm where waste dumping was occurring). 

After the pilot visits were complete. National Center staff developed a profile of the 
programs, identifying issues that cut across the five programs as well as corresponding 
recommendations for program improvement. The resulting report, "Profiling Teacher 
Development Programs: A Pilot Report,'^ was drafted for review by the Labs and DOE 
in October, 1993, and completed in December, 1993. 

After the pilot, other Laboratories profiled one of their teacher development programs. 
The eight programs profiled were: the Teachers' Supercomputing Workshop, Lawrence 
Livermore National Laboratory; Fossil Energy Teacher Institutes, Oak Ridge Institute 
for Science and Education; BEAMS (Becoming Enthusiastic About Math ana Science), 
Continuous Electron Beam Accelerator Facility; Oak Ridge Materials Science Summer 
Institute, Oak Ridge National Laboratory; Inservice Course in Environmental Science, 
Brookhaven National Laboratory; Partners for Terrific Science and Chemical Education 
for Community Understanding Program (CEPUP), Feraald Environmental Management 
Project; and the National Teacher Institute in Materials Science and Technology, Pacific 
Northwest Laboratory. 
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Each Laboratory was to complete a template on its selected program, send it to the 
National Center for feedback, and then submit a revised template and set of issues and 
recommendations to the National Center staff, who in turn would write a summary 
report. In addition to the five pilot sites, only two of the remaining eight programs sent 
in both a revised template and set of issues and recommendations. Others completed 
parts (e.g., a template but no discussion oi" issues and recommendations). Factors that 
may have contributed to this lack of completion are: (1) insufficient time and attention 
given to building the skills and knowledge base necessary for engaging in this type of 
self-evaluation, (2) the format of the template which constrains in-depth responses, and 
(3) the lack of good examples of template and profiling completions to serve as models. 
Our current work with the Laboratories is addressing these issues. 

Some Emerging Results 

1. Viewing Teacher Development as a Long-Term Proposition 

The research literature on teacher development points out the need for sustained 
engagement with teachers; "one-shot" efforts have limited usefulness (FuUan, 1991; 
Loucks-Horsley et al., 1987). For teachers to make meaningful change in their practice, 
a development effort must be both intensive and extensive. Indeed, the majority of the 
Lab programs that were profiled provided participants with intense experiences: a three 
week residential program, often with some follow up, was typical. The most intensive 
was the program that worked with the same teachers over a three-year period during the 
summer and academic year. There were exceptions, however. One was a program that 
included one-day inservices for teachers to learn to use activities developed by other 
teachers to accompany field trips and videos. The shortest program - designed to train 
participants in the use of a kit of "^terials from Lawrence Hall of Science - was five 
hours. 

When Laboratories consider alternative designs that allow their teacher development 
activities to extend over time, they are faced with some difficult issues. First is one of 
resources. Having one-week sessions spread out over the year rather than one three- 
week session raises travel costs. Another issue is teacher time: Labs have learned that 
summer sessions are preferable to those that take teachers out of class during the school 
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year. Finally, related to the first issue of resources, is the number of teachers that can be 
involved. If more time is taken per teacher, necessarily the number of teachers reached 
will need to decrease, 

2. Helping Teachers Make Explicit Links with Their Own Curriculum 

The majority of programs that were profiled provided teachers with materials or the 
opportunity develop lessons, units, or activities to use back home; yet only two made any 
atten.pt to make explicit links between the activities and the curriculum frameworks of 
the state and/or city where teachers worked. With the current emphasis on standards 
and frameworks, teachers are nearly all pressured to teach within some parameters; and 
if they are to use what they learn in Laboratory programs, Labs need to pay attention to 
these frameworks in their sessions. 

Good frameworks define not only content, but they have a teaching/learning model that 
is based on research (see reports from the National Center for Improving Science 
Education). Most frameworks that are emerging at this time (as opposed to some 
written in the '70s and '80s) are similar in their focus on major concepts (such as change, 
systems, cause and effect), skills and dispositions toward science; and their learning 
models resemble the inquiry or investigative process, with attention to presenting content 
in a context appropriate to students' development levels. Helping teachers think about 
their instruction within such a framework is very helpful, particularly if it is one that 
guides their district's or state's science curriculum. Casting new material within an 
existing context makes it seem more integrated than something completely different that 
has been added on. For urograms interested in having teachers transfer their Lab 
experience to the classroom, this is an important point to keep in mind. The more 
integrated the change, the more easily can it be implemented. 

3. Making Links to the Community and C ultural Context 

To the extent that DOE is serious about its commitment to reaching students from 
underrepresented populations. Labs need to consider how they work with teachers who 
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themselves are from these populations and/or who teach students from these populations. 
This issue cuts two ways. First is the hope that communities can become advocates for 
science and technology education, supporting their teachers in efforts to improve as well 
as becoming actively involved in helping students learn. Second is the need for teachers 
to undei tand the cultural context of the students they teach along with differences in 
their learning style preferences and, more relevant to Lab programs, to understand the 
way the students' culture views Western science and the implications that has for student 
learning. 

To their credit, the Laboratories exert considerable effort to recruit teachers serving 
minority student populations. With some exceptions, however, the profiling data indicate 
that most programs are not specifically addressing the issue of cultural relevance of 
science for the particular populations the teachers were teaching, including high 
concentrations of African American, Hispanic, and Native Ajnerican students. A random 
classroom observation made in one program showed that boys were actively involved in 
the lessons but girls were not. No program specifically mentioned addressing the needs 
and interests of girls. Descriptions focused largely on the presence of role models 
(females and minorities) rather than the content and/or presentation of the material. 
One program indicated that a session on multicultural education was part of their 
institute but did not say what specific content was covered. A few programs said that 
their program accounted for diversity but provided no details as to how that was 
occurring. 

4. Involving Schools or Districts to Support Teachers and Extend Their Reach 

Teachers need support back home to do new things, including a relationship with the 
principal or department chair that creates realistic expectations, encourages specific kinds 
of help after the institute, and, expresses the administrator's commitment to the teacher*s 
success in implementing changes. Support is even more important when Lab programs 
hope for change beyond the individual teacher, as was the case for most of the programs 
profiled. In eight of the programs there was an expectation that the teachers would 
share the information and experience gained during the Lab program with their 
colleagues. 
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Attending a DOE Lab workshop or some form of teacher inservice requires 
administrator approval, but more than that is needed to help the teachers work with 
their colleagues. Spreading their new knowledge and skills requires time, making science 
a priority in the school, supplies, and understanding that all won't be changed effectively 
overnight. Moreover, the teacher presenting a workshop for their colleagues may need 
assistance in the design of the workshop, presentation and group dynamics skills, 
audiovisual materials, evaluation, follow-up, and other aspects of planning and 
conducting staff development. The Labs might play a useful role in communicating with 
their teacher participants via electronic mail in support of teachers' dissemination 
activities. 



5. Advocating a New Model of Pedagogy 

DOE Laboratories have a clear contribution to make in terms of science content: they 
are places where scientific knowledge is not only used day-to-day but refined and 
created. Thus a logical goal for teacher development programs is updating teachers' 
science content knowledge. Less clear, however, is the Lab's role in promoting new ways 
of teaching science. 

Laboratories' programs varied widely in their approach to pedagogy. One program 
purposely did not model particular teaching principles or strategies, instead taking the 
approach of a university seminar enriched by tours and demonstrations. Other programs 
were also more traditional in that the typical format was lecture/discussion/ 
demonstration/lab. Field trips and tours were common components of these programs. 
Other programs tried using more activity-based approaches with varying degrees of 
success. Participants in one program said of their hands-on, inquiry based approach that 
they used the tools and processes of scientists and were immersed in the scientific 
process, having designed and conducted experiments to which they did not know the 
answers at the outset. Other programs used activity-based approaches with apparently 
insufficient understanding of the specifics of such approaches, especially how to ensure 
that teachers were increasing their science knowledge as well as process skills. 
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6. The Role of Labs in Teacher Development 



The Department of Energy is a relatively new player in teacher development, and it is 
continuously searching for its optimal role. The profiled teacher development programs 
were taking different approaches to a given Lab's role. Some of the Labs clearly lead 
from the strength of the Lab in linking their teacher development program to their 
ongoing research. Others use scientists to demonstrate that science can be relevant, fun, 
and a career pursuit of "real people." Still others engage teachers with scientists in 
investigations like those done at the Lab, in an effort to model how science learning can 
be holistic, real-world, engaging, and relevant. Some attempt to demonstrate how 
science, technology, and mathematics are integrated - rather than separated into 
distinct areas in real-life problems. 

All these are critically important roles for scientists - to connect teachers to the real 
world and knowledge base of science, and to help them develop a deep understanding of 
how scientific knowledge is created. This increases the potential that students will not 
think of science as a dry body of facts to be mastered, a field they would never want to 
enter. 

Having Lab staff demonstrate science teaching approaches, whether with teachers in 
their institutes, or with students in their classrooms, is another role altogether. 
Acknowledging that teaching, as a profession, has its own knowledge base and technical 
skills suggests that scientists should not be expected to be excellent teachers (just as 
teachers shouldn't expected to be excellent scientists). This issue for the Labs is whether 
they believe that their institutes should exemplify cutting-edge teaching strategies as well 
as cutting-edge science and, if so, how they can do that. 
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